Advanced touch control of interactive map viewing via finger angle using a high dimensional touchpad (HDTP) touch user interface

ABSTRACT

High Dimensional Touchpad (HDTP) or other user interface technology implemented in touch screens used on computers, smartphones, or other mobile devices provides advanced touch control of a variety of interactive map applications using one or more of a user&#39;s finger position or movement in one or more of the roll angle, pitch angle, yaw angle, and downward pressure directions. Implementations also can be responsive to a user&#39;s finger position or movement in the left-right and forward-backward directions. Implementations can also use HDTP or other user interface technology implemented on the back of a mouse. Also, the interactive map imaging application may use a connection over the internet or other network.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/511,930, filed Jul. 29, 2009, which is a continuation-in-part of U.S.application Ser. No. 12/502,230, filed Jul. 13, 2009, which claims thebenefit from U.S. Provisional Application No. 61/080,232, filed Jul. 12,2008, the contents of which are hereby incorporated by reference hereinin their entirety.

FIELD OF THE INVENTION

This invention relates to the use of a High Dimensional Touchpad (HDTP)providing enhanced parameter capabilities to the control computer windowsystems, computer applications, web applications, and mobile devices, byusing finger positions and motions comprising left-right,forward-backward, roll, pitch, yaw, and downward pressure of one or morefingers and/or other parts of a hand in contact with the HDTP touchpadsurface.

DESCRIPTION OF THE RELATED ART

The incorporation of the system and method of the invention allows forenhanced control of at least computer window systems, computerapplications, web applications, and mobile devices. The inclusion of atleast one of roll, pitch, yaw, and downward pressure of the finger incontact with the touchpad allows more than two user interface parametersto be simultaneously adjusted in an interactive manner. Contact withmore than one finger at a time, with other parts of the hand, and theuse of gestures, grammar, and syntax further enhance these capabilities.

The invention employs an HDTP such as that taught in issued U.S. Pat.No. 6,570,078, and U.S. patent application Ser. Nos. 11/761,978 and12/418,605 to provide easier control of application and window systemparameters. An HDTP allows for smoother continuous and simultaneouscontrol of many more interactive when compared to a mouse scroll wheelmouse. Tilting, rolling, or rotating a finger is easier than repeatedlyclicking a mouse button through layers of menus and dialog boxes ordragging and clicking a button or a key on the keyboard. Naturalmetaphors simplify controls that are used to require a complicatedsequence of actions.

SUMMARY OF THE INVENTION

In an embodiment, the invention includes a system and method forcontrolling an electronic game, the method comprising touching atouchpad with at least one finger, measuring at least one change in oneangle of the position of the finger with respect to the surface of thetouchpad and producing a measured-angle value, and using themeasured-angle value to control the value of at least one user interfaceparameter of the electronic game.

In an embodiment, the invention includes a system and method forcontrolling a polyhedral menu, the method comprising touching a touchpadwith at least one finger, measuring at least one change in one angle ofthe position of the finger with respect to the surface of the touchpadand producing a measured-angle value, and using the measured-angle valueto control the value of at least one user interface parameter of thepolyhedral menu.

In an embodiment, the invention includes a system and method forcontrolling a computer operating system, the method comprising touchinga touchpad with at least one finger, measuring at least one change inone angle of the position of the finger with respect to the surface ofthe touchpad and producing a measured-angle value, and using themeasured-angle value to control the value of at least one user interfaceparameter for controlling the computer operating system.

In an embodiment, the invention includes a system and method forcontrolling the observation viewpoint of a three-dimensional (3D) map,the method comprising touching a touchpad with at least one finger,measuring at least one change in one angle of the position of the fingerwith respect to the surface of the touchpad and producing ameasured-angle value, and using the measured-angle value to control thevalue of at least one user interface parameter for controlling theobservation viewpoint of the 3D map.

In an embodiment, the invention includes a system and method forcontrolling the observation viewpoint of a surrounding photographicemersion, the method comprising touching a touchpad with at least onefinger, measuring at least one change in one angle of the position ofthe finger with respect to the surface of the touchpad and producing ameasured-angle value, and using the measured-angle value to control thevalue of at least one user interface parameter for controlling theobservation viewpoint of the surrounding photographic emersion.

In an embodiment, the invention includes a system and method forcontrolling the orientation of a simulated vehicle, the methodcomprising touching a touchpad with at least one finger, measuring atleast one change in one angle of the position of the finger with respectto the surface of the touchpad and producing a measured-angle value, andusing the measured-angle value to control the value of at least one userinterface parameter for controlling the orientation of a simulatedvehicle.

In an embodiment, the invention includes a system and method forcontrolling the rotational angle of a graphical object, the methodcomprising touching a touchpad with at least one finger, measuring atleast one change in one angle of the position of the finger with respectto the surface of the touchpad and producing a measured-angle value, andusing the measured-angle value to control the value of at least one userinterface parameter for controlling the rotational angle of a graphicalobject

The invention will be described in greater detail below with referenceto the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention andtogether with the description serve to explain the principles of theinvention.

FIGS. 1 a-1 j illustrate exemplary arrangements and physical formatsemploying an HDTP touchpad. The exemplary component tactile imagesensor, interface electronics, and a processor may be includedcollectively as components of laptop computers, mobile phones, mobiledevices, remote control devices, etc.

FIG. 2 a depicts an exemplary realization wherein a tactile sensor arrayis provided with real-time or near-real-time data acquisitioncapabilities.

FIGS. 2 b and 2 c illustrate exemplary data flows in an embodiment of anHDTP system.

FIGS. 3 a-3 f illustrate exemplary six parameters that can beindependently controlled by the user and subsequently recorded byalgorithmic processing as provided by the invention.

FIG. 4 illustrates how a finger can simultaneously adjust several or allof the parameters with viable degrees of independent control.

FIG. 5 illustrates an exemplary embodiment wherein parameters, rates,and symbols may be generated in response to the user's contact with atactile sensor array, which in turn may be interpreted as parameterizedpostures, gestures, parameterized gestures, etc.

FIGS. 6 a-6 d depict exemplary operations acting on various parameters,rates, and symbols to produce other parameters, rates, and symbols,including operations such as sample/hold, interpretation, context, etc.,which in turn may be used to implement parameterized further details ofpostures, gestures, parameterized gestures, etc. and their use bysystems and applications.

FIG. 6 e shows an exemplary embodiment wherein some parameters andevents are tapped and used for focus control and selection.

FIG. 7 illustrates an exemplary polyhedron desktop featured by somecontemporary operating systems.

FIG. 8 illustrates an exemplary feature of some operating systems thatshows a preview of each open window.

FIG. 9 illustrates an exemplary set of file folders visited by filebrowser and the direction of flow in the browse history.

FIGS. 10 a-10 d depict exemplary file browser windows whose dimension iscontrolled by interpreted gestures of a user.

FIGS. 11 a-11 c illustrate exemplary file browser windows, comprisingvarious sizes of icons, which can be controlled by interpreted gesturesmade by a user.

FIGS. 12 a-12 d illustrate exemplary internet browser windows.

FIG. 13 a illustrates an exemplary internet browser window with a wordhighlighted function invoked by a user.

FIG. 13 b illustrates an exemplary internet browser window displayingthe definition of the highlighted word in FIG. 13 a.

FIG. 14 illustrates an exemplary set of previously visited webpages andthe direction of flow in the browsing history.

FIG. 15 a illustrates an exemplary initial screen view of a geographicinformation program.

FIG. 15 b illustrates an exemplary screen view with adjusted observationpoint.

FIGS. 16 a and 16 b illustrate exemplary screen views of geographicinformation system with varied vertical observation points.

FIGS. 17 a-17 c illustrate exemplary screen views of geographicinformation system with varied horizontal observation points.

FIG. 18 a illustrates an exemplary screen view of a web mapping serviceapplication.

FIG. 18 b illustrates an exemplary screen view of a web mapping serviceapplication with a feature that displays panoramic views from a positionon the map.

FIGS. 18 c-18 e illustrate exemplary screen views of a web mappingservice application with a feature that displays panoramic views alongthe street.

FIGS. 19 a-19 c illustrate exemplary screen views of a flight simulatorgame where the view from an aircraft is pitched upward or downward.

FIGS. 20 a-20 c illustrate exemplary screen views of a flight simulatorgame where the vertical orientation of an aircraft is rolledcounter-clockwise or clockwise.

FIG. 21 a illustrates an exemplary screen view of a first-person shootergame.

FIG. 21 b illustrates an exemplary screen view of a weapon selectionwindow of a first-person shooter game.

FIG. 22 illustrates an exemplary application of an object being rotatedby interpreted gestures of a user in a computer aided design/draftingapplication.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides for the control of computer windowsystems, computer applications, and web applications using an HDTP inuser interfaces that capture not only left-right and forward-backpositions of a finger in contact with the touchpad surface but alsofinger motions and positions comprising roll, pitch, yaw, and downwardpressure of the finger in contact with the touchpad.

FIGS. 1 a-1 j illustrate exemplary setup physical formats employing suchan HDTP system. In some embodiments, such an HDTP system comprises of atactile sensor array, interface electronics, and at least one processor.

An exemplary tactile sensor array embodiment may comprise regular arraysof pressure-measurement, proximity-measurement, optical-measurement, orother measurement elements or cells. However, other types of sensorsadapted to sense at least a tactile image, a pressure image, a proximityimage, or an optical image resulting from a finger, multiple fingers,and/or other hand contact with the touchpad are also provided for by theinvention.

As shown in FIG. 1 a, exemplary interface hardware may provideassociated controls and/or visual indicators or displays. Alternatively,as illustrated in FIG. 1 b, associated controls may be part of aGraphical User Interface (GUI) operating on the associated computer oron other articles of equipment. A tactile image sensor system andassociated interface hardware also may be configured to share the samehousing with the system or portions of it as shown in FIG. 1 c. Thetactile image sensor system, interface electronics, and a processor mayalso be configured to share a common housing environment as shown inFIG. 1 d. A tactile image sensor system can be a part of mobile deviceas shown in FIG. 1 e, and such device can be configured to work as aremote control system as shown in FIG. 1 f. In an embodiment, sensorarray and interface hardware may be implemented as a touchpad modulewithin a laptop or a personal computer as shown in FIG. 1 e. The tactilesensor array may be configured to be used as a touchpad elementincorporated into a handheld device, such as a field measurementinstrument, bench test instrument, Personal Digital Appliance (PDA),cellular phone, signature device, etc. An exemplary depiction of anotherexemplary handheld device, as may be used in commerce, services, orindustry, is shown in FIG. 1 f. A tactile image sensor system can beadded to the back of a mouse, for example as in depicted in FIGS. 1 g-1j.

In an exemplary embodiment, a tactile image sensor system comprises atactile sensor which in turn comprises an array of tactile measurementcells. Each tactile measurement cell detects a measurable tactilequantity as a numerical value, and interface hardware sends suchnumerical data to the processor where the data are processed andtransformed into information describing the position and movement of afinger, multiple fingers, or other part of the hand, etc.

A key feature of the touchpad HDTP is its capability to process andextract values of parameters from tactile images in real-time or nearreal-time. FIG. 2 a illustrates an exemplary dataflow embodiment. Inthis example, the tactile image sensor system may be periodicallyscanned or otherwise produce an ongoing sequence or snapshot of tactileimages. In analogy with visual images, each such tactile image in thesequence may be called a “frame.” In this example, each frame isdirected to image analysis software where algorithms and/or hardware areused to calculate or extracts a number of parameters associated withhand contact attributes of the tactile image frame.

FIG. 2 b illustrates a first exemplary data flow in an embodiment of anHDTP system. Here, a Tactile Image Sensing element provides real-timetactile image data. In some embodiments, this real-time tactile imagedata may be advantageously organized in a pre-defined manner, forexample as an ongoing sequence of “frames” similar to those comprised bymotion video).

The real-time tactile image data is presented to an Image Process andAnalysis element such as those in the previously cited patents and/orthose to be described later. The Image Process and Analysis element maybe configured to responsively produce values or streams of values of rawparameters and symbols. In some embodiments, these raw parameters andsymbols do not necessarily have any intrinsic interpretation relative tospecific applications, systems, or a computer operating system. In otherembodiments, the raw parameters and symbols may in-part or in-full haveintrinsic interpretation. In embodiments where raw parameters andsymbols do not have an intrinsic interpretation relative toapplications, a system, or a computer operating system, the rawparameters may be presented to an Application Specific Mapping element.Such an Application Specific Mapping element may responsively produceApplication-Specific parameters and symbols that are directed to aTarget System or Application.

In some multi-application situations or embodiments, some raw parametersand symbols may be assigned or interpreted in a universal or globallyapplicable way while other raw parameters and symbols may be assigned orinterpreted in an application-specific manner. FIG. 2 c illustrates asecond exemplary data flow in an embodiment of an HDTP system whichincorporates this consideration. Here, the raw parameters and symbolsmay be directed to a both a Global or Universal Mapping element and anApplication Specific Mapping element. The output of each of theseelements is directed to a Target System or Application as directed by afocus selection element (for example, as found in a computer windowingsystem). The same focus selection element may also be used tosimultaneously direct the raw parameters and symbols to a particularApplication Specific Mapping element that is associated with the TargetSystem or Application.

Many variations, combinations, and reorganizations of these operationsand concepts are possible as is clear to one skilled in the art. Suchvariations, combinations, and reorganizations of these operations andconcepts are provided for by the invention.

FIGS. 3 a-3 f illustrate six exemplary parameters that can beindependently controlled by the user and subsequently recorded byalgorithmic processing as provided for by invention. These six exemplaryparameters are:

-   -   left-right translation (FIG. 3 a), sometimes referred as “sway;”    -   forward-back translation (FIG. 3 b), sometimes referred as        “surge;”    -   more-less downward pressure (FIG. 3 c), sometimes referred to as        “heave;”    -   rotation (FIG. 3 d), sometimes referred to as “yaw;”    -   left-right tilt (FIG. 3 e), sometimes referred to as “roll;”    -   forward-backward tilt (FIG. 3 f), sometimes referred to as        “pitch.”

These parameters may be adjusted individually, in sequence, orsimultaneously. Combining these parameters allow numerous degrees offreedom. As demonstrated in FIG. 4, the finger 400 can readily,interactively, and simultaneously adjust several or all of theparameters simultaneously and with viable degrees of independentcontrol.

FIG. 5 illustrates an exemplary embodiment which can transform simplecontact with (or other operative stimulus of) the sensor array into arich information flux of parameter, rate, and symbol values. Togetherwith the rich metaphors available with the touch interface, a tremendousrange of synergistic user interface opportunities can be provided by theHDTP. The rich information flux of parameter, rate, and symbol values inturn may be interpreted as parameterized postures, gestures,parameterized gestures, etc. as may be advantageous for a system and/orapplications.

The HDTP provides for additional capabilities. For example, a sequenceof symbols may be directed to a state machine, as shown in FIG. 6 a, toproduce other symbols that serve as interpretations of one or morepossible symbol sequences. In an embodiment, one or more symbols may bedesignated to carry the meaning of an “Enter” key, permitting forsampling one or more varying parameter, rate, and/or symbol values andholding the value(s) until, for example, another “Enter” event, thusproducing sustained values as illustrated in FIG. 6 b. In an embodiment,one or more symbols may be designated as setting a context forinterpretation or operation and thus control mapping and/or assignmentoperations on parameter, rate, and/or symbol values as shown in FIG. 6c. The operations associated with FIGS. 6 a-6 c may be combined toprovide still further capabilities. For example, the exemplaryarrangement of FIG. 6 d shows mapping and/or assignment operations thatfeed an interpretation state machine which in turn controls mappingand/or assignment operations. In implementations where context isinvolved, such as in arrangements such as those depicted in FIGS. 6 b-6d, the invention provides for both context-oriented and context-freeproduction of parameter, rate, and symbol values. The parallelproduction of context-oriented and context-free values may be useful todrive multiple applications simultaneously, for data recording,diagnostics, user feedback, and a variety of other uses. All of these beused to implement parameterized further details of postures, gestures,parameterized gestures, etc. and their use by systems and applications.

In an embodiment, the measured parameters, derived by the tactile imagedata, can be either used directly or transformed into other types ofcontrol signals. The tactile image data can also be presented to shapeand image recognition processing. This could be done in post-scancomputation although aspects could be performed during scanning in someembodiments. In some implementations, shape and/or image recognition maybe applied to interpreting the tactile image measurements. In otherembodiments, shape and/or image recognition may be used to assist withor even implement tactile image measurements.

In each of the exemplary applications described below, the inventionprovides for any of the cited example postures and gestures to beinterchanged with others as may be advantageous in an implementation.

Focus Control

In many systems, especially ones comprising multiple applications ordiverse data-entry mechanisms, the information stream produced by amHDTP may need to be selectively directed to a specific application orwindow. In such systems, it may be useful to use some of the informationproduced by the HDTP for controlling which destination other informationproduced by the HDTP is to be directed to. As mentioned earlier inconjunction with FIG. 2 c, these functions are referred to as focuscontrol and focus selection.

As an example, FIG. 6 e shows an HDTP system directing an informationstream comprising for example of parameters, rates, and symbols to aFocus Selection element under the control of Focus Control element. TheFocus Control element uses a selected subset of the information streamprovided by the HDTP to interpret the user's intention for the directionof focus among several windows, applications, etc. The figure shows onlyapplications, but some of these can be replaced with application childwindows, operating system, background window, etc. In this example,focus may be controlled by an {x,y} location threshold test and a“select” symbol event, although other information may be used in itsplace.

Gestures

A gesture refers to motion of a finger, fingers, other part of the hand,or combinations of these used to direct the system with commands.Gesture recognition facilities provided by the HDTP or subsequent orassociated system may be used recognize specific changes within orbetween postures and resultantly invoke commands associated with acorresponding recognized gesture. In some embodiments, gestures may berecognized only on rates of change of underlying measured parametersprovided by the HDTP. In some embodiments, gesture recognition may alsocomprise state machines driven by threshold events in measuredparameters and/or rates of change of underlying measured parametersprovided by the HDTP.

Temporal Delimiting of Gestures

The invention provides for the system to discern and recognize anindividual gesture or a series of gestures. In such embodiments, it maybe advantageous to incorporate a time delay after user makes a gestureto enhance controllability. For example, if the system recognizes agesture and execute right away, a tap followed by rotating finger wouldbe executed as two separate events: rotate, then a single-click.

To distinguish whether a gesture is separate or part of a combinedgesture, an exemplary system may detect moments in time where there isno contact on the tactile sensor array. An exemplary system may alsodetect moments in time where there is no appreciable changes in thetactile image captured by the tactile sensor array. In an embodiment,the system may be configured to have default or user-accustomed periodof delay. In an embodiment, the system may be configured so that ifanother gesture continuously follows, then the gesture is determined tobe part of combination of gestures. In an embodiment, the system may beconfigured so that a combination and/or sequence of gestures may beviewed as another gesture. In an embodiment, the system may beconfigured so that a combination and/or sequence of gestures may beviewed as a sentence of gestures. In an embodiment, the system may beconfigured so that a combination and/or sequence of gestures is subjectto syntax and/or grammar constraints. In an embodiment, the system maybe configured so that if the gesture is followed by non-contact, thegesture is determined to be independent and corresponding action is tobe taken.

Global (Universal) and Context-Specific Gestures

Some of the gestures may be used as global commands; commands that arecommon across applications or the system. These commands include but arenot limited to opening, closing, and switching between applications,opening a windows task manager, save, print, undo, redo, copy, cut, orpaste (similar to commands by control key, Windows™ key, function keys,or Apple™ command key). Usually these controls are also provided byapplication specific menus within a specific application. Applicationsmay assign unique gestures that are recognized only within the specificapplication. While the system is being used to control specific taskswithin applications, it can be interrupted to control the whole systemwhen global gestures are recognized. When a global gesture isrecognized, it is executed regardless of which specific application isfocused. When an application specific gesture is recognized, it will beinterpreted within the application that has current focus.

In some embodiments, more complex or rarely used gestures (as opposed tosimpler or more primitive gestures) may be advantageously assigned toact as global gestures. A rationale for this is that there is far lesslikelihood that a simple gesture would be misinterpreted as a complexgesture than a complex gesture being misinterpreted as a simplergesture. Similarly, although sometimes three-finger posture or complexthree-finger movement may be interpreted as three separate one-fingerpostures or gestures, an HDTP system will not confuse one-finger gesturefor a three finger gesture.

Some context commands or application specific commands can be moreeasily be undone than some global commands. In many embodiments,misinterpreting some global commands as context command may be lesstroublesome than context commands being misinterpreted as globalcommand. Additionally, it is in many cases more complicated to undopreviously executed global commands. For example, documents that areoverwritten by accidental saving are hard to retrieve; it is timeconsuming to re-open an application that was accidentally closed;accidental print jobs sent are troublesome to stop. Moreover, assigningmore complex gestures as global, more degrees of freedom can beavailable for context gestures.

Exemplary Global Command Gestures

In an exemplary embodiment, a task manager can be opened by a uniquegesture. For example, the user may press downward with three fingers atonce, or bringing three spread fingers together. Other exemplaryembodiments may include the following “Global” or “Universal” commandsthat can be rendered while the focus is directed to a particularapplication:

-   -   To open a new document, the user can drag two fingers to the        right;    -   To close an open document, the user can drag two fingers to the        left;    -   To save an open document, the user can roll the finger to the        right, bring it to the center, and roll the finger to the left.    -   An undo command can be implemented by rotating a finger        counter-clockwise and tapping with two other fingers;    -   A redo command can be implemented by rotating a finger clockwise        and tapping with two other fingers.    -   A copy command can be implemented by pitching a finger up and        tapping with another finger;    -   A cut command can be implemented by pitching a finger up and        tapping with two other finger;    -   A paste command can be implemented by pitching a finger down and        tapping with another finger.    -   A print command can be implemented by applying pressure on the        HDTP with two fingers and tap with another finger.

Alternate assignments of various postures and gestures to such “Global”or “Universal” commands may be used as is clear to one skilled in theart.

Magnification Control

As another exemplary embodiment, a magnifying tool in text or designdocuments, a user can select an area to be magnified by settinghorizontal and vertical area by dragging two finger diagonally across,pitch both fingers forward to view the magnified view of the selectedarea, and release the fingers to return to normal view. Other metaphors,such as finger spread, may also be used.

3D-Polyhedral Menus and Pallets

The natural 3D and 6D metaphors afforded by the HDTP system provide avery natural match for the “3D-Cube” style menus, file browsers, anddesktops that are appearing in contemporary and progressive operatingsystems. For example, one or more of roll, pitch, and yaw angles may beused to rotate 3-D objects such as cubes and other polyhedron(tetrahedrons, cubes, octahedrons, dodecahedrons, etc.). The inventionprovides for polyhedra surfaces to be used for menus, browsers,desktops, pallets, and other spatial-metaphor object display andselection utilities, and for these polyhedra to be manipulated by 3Dand/or 6D actions invoked from the HDTP. The invention further providesfor these polyhedra to be manipulated by 3D and/or 6D metaphors naturalto the HDTP such as roll, pitch, yaw and also including selectionthrough Euclidian spatial coordinates, i.e. one or more of x, y, ordownward pressure (p). The invention also provides for edges and/orsurfaces of the polyhedron to be distinctively visually indexed.

Operating System Interactions

Many contemporary operating systems feature 3D desktop such as that asillustrated in FIG. 7 to enable users to switch between desktops easily.A 3D object, usually a cube, whose surfaces visually represent multipledesktops, is displayed. A 3D desktop allows a user to spin a (adjustablytransparent) cube and choose any one of the displayed desktops as thecurrently active one. In an exemplary embodiment, a user can roll andpitch a finger to spin the cube and choose a surface among the 3Ddesktop surfaces. To make a selection of desktop in this example, theuser can bring up 3D desktop by tapping the touchpad with two fingersand drag to the left, roll or pitch a finger to spin the 3D desktop cubein the corresponding direction, and release the finger when the desiredsurface is in the front. The view is then switched to normal view withthe full screen of the selected desktop.

Similar to the 3D desktop feature, some operating systems displaysstacked cascading windows of all open applications to enable users toswitch between applications easily, such as Microsoft Windows Flip, asillustrated in FIG. 8. Such a desktop feature allows users to flipthrough the stack of the open windows and choose a particularapplication window. In an exemplary application, a user can pitch afinger to scroll through the open windows and release to choose thewindow that is in the front at the moment of releasing the finger.Pitching up a finger can move the cascading stack of windows in onedirection, and pitching down a finger can move the cascading stack ofthe windows in the other direction. As an example, while a user isworking on one of the open applications, the user can bring up theWindows Flip by tapping the touchpad with two fingers and drag to theright to open the Flip window and see all the open windows ofapplications, pitch a finger up or down to scroll through the cascadingwindows of open applications, and release the finger to select thedesired application window.

In another exemplary embodiment, a browser window displaying thumbnail,tiles, or icons view, a user can navigate and choose a thumbnail, tile,or icon by tilting the finger left, right, up, or down to move theselection in a corresponding direction. For example, a user can open abrowser window of default location or home directory (usually MyComputer in Microsoft Window operating system) by tapping the touchpadwith two fingers and dragging upward. As mentioned in an earliersection, rarely used gestures or gestures with more complexity are goodchoices for global gestures as misinterpretation of global commands canbe more troublesome than that misinterpretation of context orapplication command. Thus, two fingers instead of one are used here, anddragging fingers upward is used as a natural metaphor for moving up inthe hierarchy. Tilting two fingers up can open a folder one step up inthe hierarchy of current open folder and tilting two fingers downwardcan open a folder one step down in the hierarchy of current open folder.Another example is to roll two fingers to the left to go back to afolder previously visited or to roll two fingers to the right to move tothe “forward” folder. FIG. 9 illustrates how the file browser browsesthrough the history of visited folders. Elements 901-904 represent thefolders visited including the current open folder 904, 911 representsthe direction the browser will navigate the history when the user rollstwo fingers to the left to move back to the folder previously visited,and 912 represents the direction the browser will navigate the historywhen the user rolls two fingers to the right to move forward in thehistory. For example, if the user rolls two fingers to the left to goback to a folder previously visited while the file browser is displayingcontents of the folder 904, the browser will display the folder 903.Afterwards, if the user rolls two fingers to the right to go forward inthe history while the browser is displaying the contents of folder 903,the file browser will display the contents of folder 904.

In another exemplary embodiment, placing the cursor anywhere on thetitle bar of any floating file browser window and rotating a fingerclockwise can increase the size of the window. FIG. 10 b illustrates anexemplary window with increased size as compared to the windowillustrated by FIG. 10 a. Placing the cursor anywhere on the title bar1000 of any floating file browser window and rotating a fingercounter-clockwise can decrease the size of the window. FIG. 10 dillustrates an exemplary window with decreased dimensions relative tothe window illustrated by FIG. 10 c.

In another exemplary embodiment, placing the cursor on empty region ofany window and rotating a finger clockwise can be used to increase thesize of the thumbnails, tiles, or icons. Similarly, placing the cursoron empty space of any window and rotating a finger counter-clockwise candecrease the size of the thumbnails, tiles, or icons. FIG. 11 aillustrates a file browser window with icons that are smaller in sizerelative to the icons in FIG. 11 b, and FIG. 11 c illustrates a filebrowser window with icons that are larger in size relative to the iconsin FIG. 11 b. Placing the cursor on any task bar items and rotating twofingers clockwise can maximize the application window, while placing thecursor on anywhere on the title bar of any application window androtating two fingers counter-clockwise can minimize the applicationwindow. Rotating a finger clockwise and using another finger to tap canbe implemented to do the same task as the right click on a mouse. Forexample, a user can rotate a finger clockwise to open the “right-click”menu, move a finger up or down to scroll through the items in the menuappeared once the menu appears, and tap the finger to select an itemfrom the menu. Tilting a finger while the cursor is placed on a startmenu can be used to open the start menu. When the start menu is open,the user can use a finger to scroll up or down through items on the menuand tap to execute the selected item. As another exemplary application,when a multiple tab feature becomes available in file browser windows(similar to internet browsers' multiple tab feature) opening a new tabin the file browser can be implemented by a clockwise rotation of twofingers. Similarly, closing the current tab can be implemented by acounter-clockwise rotation of two fingers.

Internet Browser

Enhanced parameter capabilities allow faster internet browsing byenabling users for fast switching between webpages, shortcuts to openand close webpages, fast navigation of history of visited webpages, etc.Similar to multiple tab file browser window, a user can rotate a fingerclockwise and use another finger to tap to open a new tab 1222 forbrowsing. FIG. 12 b illustrates an exemplary internet browser windowwith an additional tap 1222 with initial tab 1221 open. While multipletabs 1241-1245 are open, a user can rotate the finger counter-clockwiseand use another finger to tap to close the tab 1245 that currently hasfocus in. FIG. 12 d illustrates tabs 1241-1244 remaining after the tab1245 is closed. In FIG. 13 a and FIG. 13 b, a user can also drag afinger across a word 1301 to select the word, and roll the finger to theright and use another finger to tap to have the browser look up thedefinition of the word in an online dictionary website. FIG. 13 billustrates a new tab 1311 with the page that is displaying thedefinition of the word 1301 user selected.

Another example is to roll the finger to the left while dragging thesame finger to the left to go back to a webpage previously visited or toroll a finger to the right while dragging the same finger to the rightto move to the “forward” page. FIG. 14 illustrates how the navigatorbrowses through the history of visited webpages. 1401-1404 represent thewebpages visited including the current page 1404, 1411 represents thedirection the browser will navigate history when the user rolls thefinger to the left while dragging the same finger to the left to go backto a webpage previously visited, and 1412 represents the direction thebrowser will navigate history when the user rolls the finger to theright while dragging the same finger to the right to go forward in thehistory. For example, if the user rolls the finger to the left whiledragging the same finger to the left to go back to a webpage previouslyvisited while the browser is displaying the webpage 1404, the browserwill display the webpage 1403. Afterwards, if the user rolls the fingerto the right while dragging the same finger to the right to go forwardin the history while the browser is displaying the webpage 1403, thebrowser will display the webpage 1404.

As another exemplary embodiment, user can shift the focus among opentabs in a browser by rotating a finger. When there are multiple opentabs in a browser, the user can rotate a finger while the cursor isplaced on one of the open tabs to scroll through the tabs and select atab to be displayed in the browser.

Navigation Applications

In geographic information systems, such as maps land by superimpositionof images, there are separate controls for switching observation pointsuch as zooming, panning, horizontal direction, or vertical direction.These controls can be combined into simple and easy motions, and havingnatural metaphors as control avoids conflicts among integratedapplications. In an exemplary application, a user can pan or drag themap to the left or right, up, or down by dragging a finger on thetouchpad in the corresponding direction. For example, when a user placesa finger on the map and drag the finger to the left, the area of the mapshowing will be shifted to the right, so more of the right side of themap will be displayed. Also, a user may pitch a finger up or down toshift the viewpoint up or down. For example, as the user pitches thefinger up, what the user sees will be as if the user was looking at thegeographical image from higher up. A user can also pitch two fingers upor down to zoom in on a map or zoom out. For example, when the userpitch two fingers up to zoom in on a map, the application will show acloser view of the horizon or objects, and when the user pitch twofingers down to zoom out, the application will show a broader view.Rotating a finger clockwise or counter-clockwise can rotate theviewpoint or change the direction of the view left or right. FIGS. 17a-17 c illustrate exemplary views varying the horizontal direction ofthe viewpoint. Rotating a finger clockwise to rotate the view point tothe left will generate view as if the user turned to the right, androtating a finger counter-clockwise to rotate the viewpoint to the rightwill generate view as if the user turned to the left.

These controls can be combined to control more than one thing at a time.There are several possibilities; for example, when a user is pitching afinger up as the user is rotating the finger counter-clockwise, thedirection of the view will be rotated to the left as the viewpoint israised. When the user is pitching a finger downward as the user rotatesa finger clockwise, the view point is rotated to the right as the viewpoint is being lowered. This opens vast new possibilities for controlsin gaming, which will be discussed in a later section.

Web Mapping Service Applications

In web mapping service applications, similar controls can beimplemented. Since most web mapping service applications are based onground level, vertical shifting of the observation point may not beavailable, but all other controls can be implemented in the same manner.A user can pan or drag the map by dragging on the touchpad in thedesired directions, zoom in or out of the area of the map by pitchingtwo fingers upward or downward, or switch the direction of the view byrotating a finger clockwise or counter-clockwise.

In geographic information systems or web mapping service applicationswith a feature that displays panoramic surrounding photographic emersionviews from a street perspective (i.e., Google Street View), similarcontrols can be implemented. The user can move the observation pointalong the streets on the map or the image of the area by dragging afinger in the direction of desired movement, and the user can switch thedirection of the view by rotating a finger clockwise orcounter-clockwise. For a more detailed example, when a user moves afinger upward, the application will generate views as if the user iswalking forward, and when the user rotates the finger counterclockwise,the application will generate views as if the user turned to the left orto the west. FIG. 18 b illustrates an exemplary screen view of a webmapping service application with a feature that displays panoramic viewsalong the street in a window 1811. FIG. 18 d illustrates the screen viewof initial position. FIG. 18 c illustrates an exemplary screen view ofwhen the user rotates a finger to switch the view towards to the west,and FIG. 18 e illustrates an exemplary screen view of when the userrotates a finger clockwise to switch the view towards to the east. Also,in implementations where views along the street are only displayed atuser discretion, the user can enter the Street View mode by pressing onefinger down and exit from the Street View mode by pressing two fingersdown.

Computer and Video Games

As games heavily rely on 3D features more and more, these additionalparameters provided by the HDTP can be more useful as they can producecontrols using natural metaphor. Controls that previously requirecomplicated sequence of arrow keys and buttons can easily be implementedby combination of parameters.

Flight Simulator Game

For example, in a flight simulator game, controls that are similar tothose in 3D navigation applications can be used. The user can controlthe direction of the movement by rolling, pitching, or rotating thefinger. The user can control horizontal orientation of the aircraft byrolling the finger; roll the finger to the left to have the aircraftroll counter-clockwise and roll the finger to the right to have theaircraft roll clockwise. FIG. 20 a illustrates an exemplary view fromthe simulated aircraft when the aircraft is rolling to the left. Thehorizon 2011 appears tilted counter-clockwise relative to the horizontalorientation of the aircraft. FIG. 20 b illustrates an exemplary viewfrom the simulated aircraft when the aircraft is not rolling. Thehorizon 2021 appears leveled with the horizontal orientation of theaircraft. FIG. 20 c illustrates an exemplary view from the simulatedaircraft when the aircraft is rolling to the right. The horizon 2031appears tilted clockwise relative to the horizontal orientation of theaircraft. The user can control vertical orientation (or pitch) of theaircraft by pitching the finger; pitch the finger up to pitch theaircraft upward and pitch the finger down to have the aircraft downward.In a more detailed example, the simulated aircraft can take off as theuser pitches a finger downward to have the aircraft pitch upward. FIG.19 b illustrates an exemplary screen view of the initial position of anaircraft, and FIG. 19 a illustrates an exemplary view from the simulatedaircraft while headed upwards and taking off. The player can land theaircraft by pitching a finger upward to have the simulated aircraft isheaded down to the ground. FIG. 19 c illustrates an exemplary screenview as the simulated aircraft approaches the ground. As the simulatedaircraft is headed up, the player can view more of objects that arefarther away from the aircraft, and as the aircraft is headed down, theplayer can view more of objects that are closer to the aircraft. Theuser can control two-dimensional orientation of the simulated aircraftat a fixed elevation by rotating the finger; rotate the finger left tohave the aircraft head to the west (or to the left) and rotate thefinger right to have the aircraft head to the east (or to the right).Exemplary views from the aircraft with varied horizontal rotation willbe similar to the views illustrated in FIG. 17 a-c. The player can alsocombine gestures for simultaneous multiple controls. For example theuser can pitch a finger upward while rolling the finger to the left orright to control the aircraft roll to the left as the aircraft is headeddown. As another example, the user can rotate a finger counter-clockwiseas the aircraft is headed up to make the aircraft change its directionto the west while the elevation of the aircraft is rising.

Other Moving Vehicle Games

As another example, similar controls can be implemented in any racinggames of car, motorcycle, spacecraft, or other moving objects. Pitchingthe finger downward can be implemented to accelerate the car; pitchingthe finger upward can be implemented to brake with adjusted amount ofpressure; rotating the finger counterclockwise can be implemented toturn the steering wheel to the left; rotating the finger clockwise canbe implemented to turn the steering wheel to the right. As the userrotates the finger counter-clockwise to turn the vehicle to the left andtilt the finger to the left, the car can drift.

Winter Sport Games

In skiing, snowboarding, or any first-person snow sports games, the usercan rotate the finger clockwise or counter-clockwise to switch thedirection; the user can roll the finger left or right to switch thecenter of weight of the body left or right; the user can pitch thefinger forward or backward to switch the center of weight of the body toaccelerate or slow down; When the skier or snowboarder hits a slightuphill or mogul, the player can jump while controlling the intensity ofthe jump by combination of speed and the degree of pitching the fingerbackward.

Summer Sport Games

In sports games where the players hit balls, such as baseball, tennis,golf, or soccer, not only the player can control the direction ofhitting, the player can also control the intensity of the hits at thesame time by combining rotation and pitching of the finger.

Shooter Games

In first-person shooter video games, the direction of player's motioncan be controlled by rotating a finger, the speed of running can becontrolled by applying more or less pressure to the finger. FIG. 21 aillustrates an exemplary screen view of a first-person shooter game. Inaddition, weapon selection window can be opened by pitching two fingersforward, and once the window is open, the player can roll the finger toscroll through the selection of weapons and release the finger to selectthe highlighted weapon and close the weapon selection window. FIG. 21 billustrates an exemplary screen view of a weapon selection window of afirst-person shooter game. Both FIG. 21 a and FIG. 21 b have beenobtained from video games that are available on the web for freedownloading.

Music Performance Experience Games

In video games where players play instruments, heave and pitch offingers can control how hard a string of an instrument is strummed orplucked or intensity of sound generated.

Media Players

In a media player, such as Winamp, Real, or Windows Media Player,increasing or reducing the volume can be implemented by pitching afinger upward or downward on the “frame.” Pitching the finger on theplaylist window, a user can scroll through the tracks and select thedesired track to be played. In an embodiment, a media player thatfeatures polyhedron menu of multiple play lists can be controlledsimilar to 3D desktop. A user can tap on the play list cube and rotatethe finger left, right, up, or down to select a surface among thesurfaces that represents different play lists. Rewinding orfast-forwarding can be implemented by rolling a finger left or right onthe timeline, and the current track may be skipped by clockwise fingerrotation and the current track may be returned to the beginning bycounter-clockwise finger rotation.

Spreadsheets

Similar to selecting a thumbnail, tile, or icon in an explorer window inan embodiment, a user can scroll through cells on a spreadsheet bytilting the finger left, right, up, or down. A user also can tap on acell in an initial position, drag the finger down to set vertical rangeof cells and drag the finger to the right to set horizontal range ofselected cells. Then the user can tilt both fingers left, right, up, ordown to move the group of selected cells. Selection of cells can be donevia different motions. For example, rolling the fingers left or rightcan select a group of multiple columns incrementally, and pitching thefingers up or down can select multiple rows incrementally.

Graphic Design Applications

As computer aided design/drafting tools features numerous features, theyprovide several menu items and options at different levels. Even insimply rotating an object or figures, there are many operations or stepsinvolved. In an exemplary embodiment, instead of moving the cursor tothe menu bar, clicking the drop-down menu to be opened, and moving themouse and clicking to select the desired function, a user can usecombined motion of rolling, pitching, rotating a finger that are easy toremember. For example, in some design applications such as AdobeFrameMaker™, in order for a user to draw a line, a user would have toselect the line tool and click on the initial and the final point with amouse every time. As an exemplary application of this invention, theuser can drag a finger on the touchpad while applying pressure on thefinger to draw a line. This way of drawing lines can be very useful whendrawing curved lines as drawing lines with a finger will draw smootherlines than lines drawn by using a mouse because drawing a curved linewith a finger will allow finer adjustments than drawing a curved linewith a hand holding a mouse.

As another example, to rotate an object, the user can click on theobject to select it and rotate the finger to rotate the object in thecorresponding direction. FIG. 22 illustrates an exemplary use of thisprocess in an exemplary application. This feature can be useful tocorrect pictures that are vertically misaligned; a user can select allof a picture and rotate the finger by desired amount of degrees. Oncethe picture is vertically aligned, the user can select the best fittingrectangular area of the picture to save. While an object is beingrotated, the user can drag the finger around to slide the object around.Recording of such motions can be useful to generate an animation of themoving object. To group objects, the user can pitch two fingers up afterthe user highlights the objects by tapping on the objects while having afinger of the other hand down on the touchpad. To increase the size of a2D or 3D object, the user can select an object and rotate two fingerscounter-clockwise to decrease the size of the object or rotate twofingers clockwise to increase the size of the object. To increase thethickness of the outline of an object, the user can tap on the outlineof the object and rotate two fingers clockwise. To decrease thethickness of the outline of an object, the user can tap on the outlineof the object and rotate two fingers clockwise. Similar implementationcan be done in word processing applications to increase or decrease thefont size. As another exemplary application, to flip an object left orright, the user can click on the object to have it highlighted, tilt afinger left or right, and tap with another finger to have the objectflipped left or right. Similarly, to flip an object towards a reflectionpoint, the user can click on the object to have it highlighted, touchthe point of reflection with a finger of the other hand, and tilt thefinger on the object towards the reflection point.

Mobile Devices

As more functionality is added as features of mobile devices, menus andcontrols for these devices become complicated. Combined motion controlbecomes extremely useful in mobile devices whose screen size is limited.Numbers of possible shortcuts increase dramatically by using combinationof motions as shortcuts. As an example of application in a mobile phone,a shortcut to “phone” or “calls” mode can be implemented bycounter-clockwise rotation of a finger, and a shortcut to applicationsmode can be implemented by clockwise rotation of a finger on the screen.For mobile devices without detection of vertical or horizontalorientation, detection method can be replaced by having the user rotatea finger on the screen. When the user wants to view pictures sideways onthe phone, the user can switch between portrait and landscape mode byrotating a finger on the screen.

As another example, while the phone is being used as music or videoplayer, the user can pitch a finger on the screen forward or backward tocontrol the volume, roll the finger left or right to rewind orfast-forward, or roll the finger left or right while dragging the fingerin the same direction to seek to the beginning of the current track orto the next track. When the mobile device is in virtual networkcomputing mode or being used as a remote control, all of the functionsdescribed so far can be implemented on the mobile devices.

Combinations of motions can also be used as identification on mobiledevices. For example, instead of methods such as entering a securitycode, a device can be programmed to recognize a series of motions asidentification. The identification process can allow users differentlevel of access, for example, calls only mode, child-proof mode, orapplication only mode. When the mobile phone receives a phone call whileit is in application mode, a user can make a global gesture to exit theapplication mode or the touchpad of the phone can be partitioned intosections so one section can control the ongoing application and theother section can control phone functionality. In general, a touchpaduser interface can be divided to have each partition control differentapplications.

Various embodiments described herein may be implemented in acomputer-readable medium using, for example, computer software,hardware, or some combination thereof. For a hardware implementation,the embodiments described herein may be implemented within one or moreapplication specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), field programmable gate arrays(FPGAs), processors, controllers, micro-controllers, microprocessors,other electronic units designed to perform the functions describedherein, or a selective combination thereof.

For a software implementation, the embodiments described herein may beimplemented with separate software modules, such as procedures andfunctions, each of which perform one or more of the functions andoperations described herein. The software codes can be implemented witha software application written in any suitable programming language andmay be stored in memory and executed by a controller or processor.

In all of the exemplary applications described, the invention providesfor any of the cited example postures and gestures to be interchangedwith others as may be advantageous in an implementation.

I claim:
 1. A method for controlling an interactive map application, themethod comprising: configuring a user interface touch sensor to beresponsive to at least one angle of contact with at least one finger,the finger belonging to a human user of a computing device and the userinterface touch sensor in communication with an operating system of thecomputing device; measuring at least one change in at least one angle ofthe position of the finger with respect to the surface of the touchsensor to produce measured data; performing real-time calculations onthe measured data to produce a measured-angle value; and using themeasured-angle value to control the value of at least one user interfaceparameter of an interactive map application; wherein at least one aspectof the interactive map application changes responsive to the angle ofthe position of the finger with respect to the surface of the touchsensor.
 2. The method of claim 1 wherein the at least one aspect of theinteractive map application comprises control of the observationviewpoint of the map.
 3. The method of claim 1 wherein the at least oneaspect of the interactive map application comprises control of thegraphically rendered map.
 4. The method of claim 1 wherein theinteractive map application comprises a network link over a network. 5.The method of claim 1, wherein the user interface touch sensor isconfigured as a touchscreen.
 6. The method of claim 1 wherein the touchsensor comprises a High Dimensional Touchpad (HDTP).
 7. The method ofclaim 1, wherein the at least one finger angle comprises the pitch ofthe finger with respect to the touch sensor.
 8. The method of claim 1,wherein the at least one finger angle comprises the roll of the fingerwith respect to the touch sensor.
 9. The method of claim 1, wherein theat least one finger angle comprises the yaw of the finger with respectto the touch sensor.
 10. The method of claim 1, wherein the userinterface touch sensor is additionally configured to be responsive tothe left-right position of the finger on the touch sensor.
 11. Themethod of claim 1, wherein the user interface touch sensor isadditionally configured to be responsive to the forward-backwardposition of the finger on the touch sensor.
 12. The method of claim 1,wherein the user interface touch sensor is additionally configured to beresponsive to at least one gesture comprising changes to left-rightposition of the finger on the touch sensor.
 13. The method of claim 1,wherein the user interface touch sensor is additionally configured to beresponsive to at least one gesture comprising changes toforward-backward position of the finger on the touch sensor.
 14. Themethod of claim 1, wherein the user interface touch sensor isadditionally configured to be responsive to heave gestures comprisingchanges to pressure applied by the finger to the touch sensor.
 15. Themethod of claim 1, wherein the user interface touch sensor isadditionally configured to be responsive to gestures comprising changesto pressure applied by the finger to the touch sensor.
 16. The method ofclaim 1, wherein the user interface touch sensor is further configuredto be responsive to simultaneous changes in at least two of finger rollangle, finger pitch angle, finger yaw angle, and finger downwardpressure.
 17. The method of claim 1, wherein the computing devicecomprises a mobile device.
 18. The method of claim 1, wherein thecomputing device is comprised by a mobile device.
 19. The method ofclaim 1, wherein the speed of the change of the finger angle is furtherused to control an aspect of the interactive map application.
 20. Themethod of claim 1, wherein at least one finger angle is used in anatural metaphor to an aspect of the interactive map application.